Stereoscopic camera apparatus

ABSTRACT

Embodiments of an apparatus for acquiring, storing, transmitting and displaying stereoscopic images are disclosed. Some of the benefits of embodiments of this apparatus include simultaneous left/right view acquisition, transmitting and displaying images from a stereoscopic camera, stereoscopic digital zooming wherein a subset of pixels is displayed, pan-tilt-zooming of the apparatus, and interactive adjustment of images. Embodiments of the disclosed apparatus are capable of producing real-time, stereoscopic image data from various illumination wavelengths, coupling to other optical instruments, changing sensor exposure parameters, storing a stereoscopic single data stream, and selectively adding a filter component.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/055,626 , filed on Feb. 28, 2016, to be issued as U.S. Pat. No.9,509,975, which is a continuation of U.S. patent application Ser. No.14/740,110 , filed on Jun. 15, 2015, now U.S. Pat. No. 9,277,203, whichis a continuation of U.S. patent application Ser. No. 13/694,782, filedon Jan. 3, 2013, now U.S. Pat. No. 9,124,877, which is a divisional ofU.S. patent application Ser. No. 11/881,617, filed on Jul. 24, 2007, nowU.S. Pat. No. 8,390,675, which is a nonprovisional application of U.S.Provisional Patent Application No. 60/833,117 filed on Jul. 24, 2006,wherein U.S. patent application Ser. No. 11/881,617 is acontinuation-in-part of U.S. patent application Ser. No. 11/256,497,filed on Oct. 21, 2005, which in turn is a nonprovisional application ofU.S. Provisional Patent Application No. 60/621,271, filed on Oct. 21,2004, wherein all of the U.S. priority applications in their entiretyare herein incorporated by reference.

BACKGROUND Description of Related Art

The use of stereoscopy, in which the user sees left- and right-eye viewsand forms a three dimensional image through stereopsis, is common inmany areas of science, medicine and entertainment. The use of an opticalinstrument to provide a stereoscopic image of objects to a user's eyesis also common. Optical instruments are used for observation,surveillance, and many other purposes.

The optical image generated by an optical instrument is typically viewedthrough eyepieces. However, the use of eyepieces in optical instrumentsystems is often problematic. Furthermore, only one observer at a timecan view images generated by the optical instrument and the observer canno longer see what is happening in the surrounding environment. Inaddition, an optical instrument , as such, cannot store images orsequences of images for later playback, process them in special ways, ortransmit them to remote viewing sites. There are also situations inwhich it is desirable to remotely view or record a stereoscopic image ofa location or object without involving a person to take the image.

Therefore, it is often desirable to use electronic imaging to acquireimages of a location, either for direct, real-time observing or forrecording Electronic imaging is a preferred method of the televisionbroadcasting, video and movie industries as well. The use of cameras andelectronic displays to acquire images is well known in the art,including the use of two cameras and a 3D display to give a stereoscopicimage.

However the two-camera systems have many disadvantages. Obtaining andmaintaining stereoscopic alignment (necessary for comfortable, long-termviewing) can be very difficult when two independent cameras are mountedon or comprise an optical instrument. The cameras generally protrudefrom the general body of the device and are often mounted in a way thatis fragile and prone to breakage. Protruding cameras can also interferewith other apparatuses in the workspace, limiting possible usageconfigurations. The two-camera systems have generally double the opticalinstrument and camera knobs and controls, resulting in an unwieldydevice difficult to operate by a single user. Dual camera systemsgenerally require numerous mounting parts, resulting in less reliabilityand more cost than a single, integrated camera.

There are also problems with mounting and connecting the cameras todisplays or storage media. The use of two cameras requires multiplecables and connectors, resulting in less reliability and more difficultinstallation than a single cable/connector arrangement of the presentinvention. The two-camera system also typically requires two cameracontrol units (CCUs) and two storage devices, and requires that they besynchronized for best image quality. This significantly increases thecost of the system.

In addition, such cameras do not allow precise positioning of theimaging sensors to each other for best stereopsis and comfortableviewing, particularly when two off-the-shelf cameras are used. Cameraswhich are wide cannot be easily positioned side-by-side with closespacing. The cameras must be individually focused after mounting, and,should adjustments such as brightness and contrast be needed, eachcamera must be controlled individually. Where the cameras containirises, they must also be individually adjusted for each camera,resulting in the potential for unequal amounts of light entering eachcamera, which can lead to difficult 3D viewing and eyestrain. All thesefactors indicate that using such a system requires skill and can be verytime-consuming.

Image processing is also problematic in such systems. The cameras mustbe electronically linked in some way so that the two image streams aresynchronized, creating additional cost and complexity. Even if the datastreams are synchronized, generally the shutters are not perfectlysynchronized such that the nth pixel from one view was not captured atthe same time as the nth pixel from the other view, causing movingobjects to show spurious parallax when displayed. Furthermore, theimages acquired by the two cameras are generally taken directly to the3D display device. Therefore, should the user require image processing,storage, transmission, or display on alternative displays, additionalprocessing units are required, creating additional cost and complexity.

The cameras used in such two-camera systems also usually conform to theNTSC or PAL video standard, both of which suffer from low resolution,poor color fidelity, and motion artifacts (due to the interlaced natureof the raster scan). Recording and editing recorded content is alsoproblematic with the two-camera system. Recorders don't generally startand end synchronously, so the two tapes or files must somehow besynchronized, resulting in additional effort and expense. Editing mayneed to be performed twice—once to each file or tape.

Information relevant to attempts to address these problems can be foundin U.S. Pat. Nos. 4,418,993; 4,583,117; 4,879,596; 4,881,122; 5,164,827;5,438,386; 6,157,337, and 6,512,892.

However, each one of these references suffers from one or more of thefollowing disadvantages: the device or system creates two independentoutput signals; is not compact; does not provide sufficient imageprocessing, recording, or transmission capability; does not haveadequate resolution in real-time for many applications; is cumbersome oris not easily operated by a single user; is large and expensive; morethan one operator is generally needed to properly control all of therequired functions in order to provide good images; it is difficult tosynchronize two separate cameras to the pixel level; two recordingdevices or image-capturing paths are required, resulting in additionalcomplexity and cost in acquiring and recording the images and editingthem as is often desirable; accessory image/data recording systems havea required start-up time prior to recording; uses significant power,requiring large batteries for mobile applications and emittingsignificant heat that could disturb sensitive environments; is morefragile than a single camera; or does not perform well if either or bothof the cameras uses automatic focusing, automatic exposure control orimage stabilization control, because such systems or devices heretoforehave not been synchronized for the two views from the two cameras;

Therefore, the use of optical instrument systems containing electroniccameras, recording devices and display therefore solves some of theeyepiece problems but creates new ones, essentially making themimpractical for routine use.

SUMMARY

Embodiments of the present invention provide improved devices andmethods for viewing and recording images and, in particular,stereoscopic images.

The embodiments of the invention relate to a compact stereoscopic cameracapable of providing visual observation and recording of images of alocation. In particular, embodiments of the present invention provide anoptical instrument having an integrated Stereoscopic Image AcquisitionDevice (SIAD) which circumvents the need for, and limitations of,eyepieces. The camera acquires and transfers high-resolution, real-timestereoscopic image data in a single data stream, from stereoscopic stillor moving views of a location or object synchronized to the pixel level,to image processing, recording, or display systems which may be includedin an integrated handheld device. The device performs the desiredfunctions without protruding elements, numerous cables and connectors,and other additional components, and could be readily operated by asingle user.

One aspect of the invention is a stereoscopic camera having an opticalinstrument and a stereoscopic image acquisition device. In oneembodiment, the camera contains mechanisms or structures designed toavoid spurious parallax. In another embodiment, the camera containsmechanisms or structures designed to control the effect of varyinginterpupillary distance (IPD). In yet another embodiment, the cameracontains mechanisms or structures designed to control the effects ofvarying convergence. In a further embodiment, the camera can provide anon-reflected view or desirable orientation of the location. In anotherembodiment, the camera contains master-slave control of adjustablechannels. In yet another embodiment, the camera may be free standing andcontains an integrated power source, image processing unit, and storagedevice. In yet another embodiment, a display mechanism is integratedinto the camera.

A second aspect of the invention is a stereoscopic camera includingmaster-slave control of adjustable camera channels such that aspects ofthe views from channels are equalized, providing optimal stereopsis.

A third aspect of the invention is a method for acquiring stereoscopicimages of a location or object, the method including steps forinteractively aligning vergence without producing substantially abrupttransitions in the views used to acquire the stereoscopic images. In oneembodiment, alignment could be achieved by a single user. In anotherembodiment, alignment could be achieved simultaneously with other cameraadjustments. In yet another embodiment, the method includes processesfor maintaining vertical position equalization in order to preventspurious parallax between the respective views.

A fourth aspect of the invention is a stereoscopic camera in which thefunctional elements of an optical instrument and a stereoscopic imageacquisition device are integrated into a single package.

A fifth aspect of the invention is an interactive method for mitigatingthe effects of camera shake while acquiring stereoscopic images.

These and other further features and advantages of the embodiments ofthe invention will be apparent to those skilled in the art from thefollowing detailed description, taken together with the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of one embodiment of the invention having astereoscopic camera system, including the system components and aviewer;

FIG. 2 is an expanded perspective view of the stereoscopic cameracomponents of the embodiment shown in FIG. 1; and

FIG. 3 is a perspective view of one embodiment of the inventioncontaining a deflecting element in addition to stereoscopic camerasystem components and a viewer.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention provides improved devices andmethods for acquiring, viewing and recording images and, in particular,stereoscopic images.

Briefly, and in general terms, embodiments of the present invention aredirected to an optical instrument having an integrated SIAD device. Inparticular, embodiments of the present invention provide an opticalinstrument having an integrated SIAD device which circumvents the needfor, and limitations of, eyepieces, and additionally includes numerousfeatures not hitherto associated with optical instruments.

In particular, embodiments of the present invention relate to a compactstereoscopic camera capable of providing visual observation of alocation. In particular, some embodiments of the present inventionprovide an optical instrument having an integrated Stereoscopic ImageAcquisition Device (SIAD) which circumvents the need for, andlimitations of, eyepieces. The camera acquires and transfershigh-resolution, real-time stereoscopic image data in a single datastream, from stereoscopic still or moving views of a location or objectsynchronized to the pixel level, to image processing, recording, ordisplay systems which may be included in an integrated handheld device.The device performs the desired functions without protruding elements,numerous cables and connectors, and other additional components, andcould be readily operated by a single user.

The following description presents embodiments of the inventionrepresenting the modes contemplated for practicing the methodsdisclosed, including the best mode. This description is not to be takenin a limiting sense but is made merely for the purpose of describing thegeneral principles of the embodiments of the invention whose scope isdefined by the appended claims.

Before addressing details of embodiments described below, some terms aredefined or clarified. As used herein, the terms “comprises,”“comprising,” “includes,” “including,” “has,” “having” or any othervariation thereof, are intended to cover a non-exclusive inclusion. Forexample, a , method, article, or apparatus that comprises a list ofelements is not necessarily limited to only those elements but mayinclude other elements not expressly listed or inherent to such process,method, article, or apparatus. Further, unless expressly stated to thecontrary, “or” refers to an inclusive or and not to an exclusive or. Forexample, a condition A or B is satisfied by any one of the following: Ais true (or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

Also, use of the “a” or “an” are employed to describe elements andcomponents of the embodiments of the invention. This is done merely forconvenience and to give a general sense of the embodiments of theinvention. This description should be read to include one or at leastone and the singular also includes the plural unless it is obvious thatit is meant otherwise.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Furthermore, any definitionsused refer to the particular embodiments described herein and are not tobe taken as limiting; the invention includes equivalents for otherundescribed embodiments. Although methods and materials similar orequivalent to those described herein can be used in the practice ortesting of the present invention, suitable methods and materials aredescribed below. All publications, patent applications, patents, andother references mentioned herein are incorporated by reference in theirentirety. In case of conflict, the present specification, includingdefinitions, will control. In addition, the materials, methods, andexamples are illustrative only and not intended to be limiting.

As used herein, the term “beam” is intended to mean a rigid member orstructure supported at one or both ends, subject to bending forces froma direction perpendicular to its length. A beam can be made flexible ina direction and rigid in others.

As used herein, the term “binocular” is intended to mean an opticalinstrument consisting of two optical paths, each comprising one or moreoptical components, such as a lens, or combination thereof for focusinga stereoscopic image of a location or object therein on, for example,the eyes of a viewer or on a sensor. A binocular can be used formagnifying a small distant object but can also be used to provide ade-magnified view of a location, for example a wide-angle stereoscopicimage of a landscape location. By comparison, a microscope generally isused for magnification of small objects which are close and heldattached to a stationary portion of the microscope to which the opticalpath is focused. A binocular is generally used to observe objects atmore varying, farther and random distances.

As used herein, the term “camera” is intended to mean a device thatconsists of one or more lightproof chambers with one or more aperturesfitted with one or more in combination lens or other optical componentthrough which the image of an object is projected and focused onto asurface for recording (as on film, for example) or for translation intoelectrical impulses or data, for display or recording (as for televisionbroadcast, for example).

As used herein, the term “camera shake” is intended to mean the effectof unintended vibration and random motion imparted to a camera by theunsteadiness of the holding device, which is generally a user's hands, avehicle's mounting bracket or an unsteady base. Camera shake appears ona display as a bouncing or vibrating view.

As used herein, the term “centration” is intended to mean the accuracywith which the optical axis of a lens in an optical instrument coincideswith the mechanical axis of a mounting in the instrument for that lens.Poor centration can cause spurious parallax when optical components aremoved relative to one another or to a sensor.

As used herein the term “channel” when referring to a stereoscopiccamera is intended to mean the components required to acquire a view ofa stereoscopic image. One nonlimiting example of a channel consists of alens, optical path and one or more sensors to acquire the view as data.Channels are typically arranged such that the optical axes are coplanarand may converge at a distant point, and the input optical componentsare generally side by side.

As used herein the term “channel spacing” or “spacing of channels”refers to the distance between the optical paths at the inputs to theinput optical components of channels, often objective lenses. Channelspacing may be adjusted in order to change one or more of the views of alocation or object, thereby providing a desired perspective of thestereoscopic image formed by the views.

As used herein, the term “controller” is intended to mean the componentof a system that contains the circuitry necessary to interpret andexecute instructions fed into the system. For example an acquisitionsystem may contain an acquisition controller. Representative graphicscontrollers include without limitation a graphics card, video card,video board, video display board, display adapter, video adapter,graphics adapter, image processing unit or combination thereof.

As used herein, the term “de-reflection” refers to reversing thereflecting effect of a deflecting element. If a mirror or otherdeflecting element is used between the object and a camera, for exampleto look around a corner, the view can be reversed electronically byreassigning the location of pixels within the views such that the viewpresented on the display is oriented as if the object was vieweddirectly.

As used herein, the term “electronic mechanisms for adjusting the sizeof an object” include automatic zoom and digital zoom.

As used herein, the term “equalize” or “equalized” is intended to meanto cause to correspond, or be like, in amount or degree as compared,including without limitation to make equal or to compensate fordifferences.

As used herein, the term “equalization of size” refers to having anobject appear at the appropriate size to each eye of the viewer.Generally for objects in front of a viewer an object will appear thesame size in each eye. As such, in a stereoscopic image it is importantthat an object appear the same size in each view in order for the viewerto form the best stereopsis. If the object does not appear to be thesame size, the size can be equalized by making the size of one viewequal to that of the other view of the object to correct the image.

As used herein, the term “equalization of vertical position”, verticalreferring to the direction perpendicular to the plane of the opticalaxes of two channels of a stereoscopic camera, refers to making anobject appear at the appropriate vertical position to each eye of theviewer. Generally a point on an object is at the same vertical positionin each view to avoid spurious parallax which detriments the viewer'sstereopsis. If the object does not appear to be at the same verticalposition, the position can be equalized by making the position of oneview equal to that of the other view of the object to correct the image.

As used herein, the term “free-standing” is intended to describe adevice which is sufficiently complete in construction such that noadditional devices are required for its operation. For example withoutlimitation, a camcorder can be free-standing as it runs on batteries andhas a built-in recording system; the user need have no other device tooperate it.

As used herein, the term “high-resolution” when referring tostereoscopic images is intended to mean at least about 1280 by 720pixels for each left or right view. It is contemplated that resolutionsof three times and eight times this minimum resolution may beimplemented depending on the state of technology for sensors anddisplays and depending on what cost is acceptable. On the other hand,the devices of the present invention may be implemented withoutlimitation with higher or lower resolutions for either one or both ofthe views.

As used herein, the term “image data” is intended to mean data producedby regular array detectors such as CMOS, CCDs and infrared devices. Thedata structures are created by the data acquisition systems oracquisition controller for use by observers, data reduction systems, andarchives.

As used herein the term “kinematic relationship” is intended to mean theability to deduce the motion of points on a device from the knowledge ofthe motion of other points on the device and the geometry of the device.For example without limitation, if one end of a long lever is lifted,say, 2 inches, it can be deduced from the kinematic relationship thatthe motion of the midpoint of the lever is about 1 inch.

As used herein, the term “lens” is intended to refer to a piece oftransparent material (such as, for example, glass) that has two oppositeregular surfaces, either both curved or one curved and the other plane,and that is used either singly or combined in an optical instrument forforming an image by focusing rays of light. “Lens” may refer to anindividual lens or a plurality of individual lenses acting incombination.

As used herein, the term “location” is intended to mean a position orsite marked by some distinguishing feature, including without limitationa place, scene, or point therein.

As used herein, the term “magnification” is intended to mean the ratioof the size of an image to the size of an object. It can be a relativeterm because an electronic image of an object imaged with a camera couldbe displayed on a large or a small display device and hence havedifferent magnifications resulting from identical image data.

As used herein, the term “mechanism” is intended to mean a process,technique, device or system for achieving a result. A mechanism may becontrolled in a variety of ways, including without limitationmechanically, electromechanically, electrically, or electronicallyoperated mechanisms.

As used herein, the term “optical” is intended to mean of or relating toor involving light or optics, including without limitation the use ofvisible radiation or combinations of visible and non-visible radiationto visualize objects.

As used herein, the term “optical component” is intended to mean a partof an optical system which deflects, refracts, restricts, focuses,manipulates, mirrors, modifies, filters or has some other intendedeffect on a beam of light including without limitation lenses, prisms,mirrors, and beamsplitters.

As used herein, the term “optical instrument” is intended to mean anyoptical instrument capable of generating images including withoutlimitation microscopes, endoscopes, binoculars, and telescopes.

As used herein, the term “optical path” is intended to mean thegenerally central ray in an optical system. Should the system have nocentral ray then the optical path is the general centerline of theaverage of all the rays.

As used herein, the term “optical device” is intended to mean any deviceor instrument capable of generating, sensing, capturing, processing,formatting, or storing images or image data.

As used herein, the phrase “real time” is intended to mean that theimage data is acquired, processed, transmitted, or displayed at asufficiently high data rate and at a sufficiently low delay that objectson a display move smoothly, for example without user-noticeable judderor latency between object motion and display motion. Typically, thisoccurs when new images are acquired, processed, and transmitted at arate of at least about 24 frames per second (fps) and displayed at arate of at least about 45 fps and when the combined processing of thesystem has no more than about 1/30^(th) sec of delay.

As used herein, the phrase “single data stream” is intended to mean acombination of more than one individual data streams into a singlestream such that a desirable aspect of the data is maintained, such astiming or scale.

As used herein, the term “sensor” is intended to mean a device thatresponds to a stimulus, such as heat, light, or pressure, and generatesone or more signals that can be measured or interpreted.

As used herein, the term “shutter” is intended to mean a cameracomponent that allows light to enter by opening and closing an aperture.

As used herein, the term “spurious parallax” is intended to mean theparallax between views in a stereoscopic image which appears imperfectto the viewer. Vertical parallax of even a small amount results in poorstereopsis. For example, spurious parallax can be caused by non-planaroptical axes of channels, unequal magnification in channels, vibrationof the camera, distorted or unequal optical paths of channels andsimilar imperfections.

As used herein, the term “stereoscopic image” is intended to mean asingle image consisting of at least two views, one corresponding to aleft-eye view, i.e. the left view, and one corresponding to a right-eyeview, the right view.

As used herein, the term “stereoscopic image acquisition device” isintended to mean a device capable of acquiring stereoscopic images froman optical instrument, or the imaging components thereof. Embodiments ofthe device acquire and transfer high-resolution, real-time image datafrom stereoscopic still or moving images to image processing, recording,or display systems. Embodiments of the device can perform the desiredfunctions without protruding elements, numerous cables and connectors,and other additional components such as eyepieces, and can be readilyadapted for use with a variety of optical instruments. Embodiments ofthe device may incorporate the functionality of related mechanisms,controllers, sensors, and processors in a non-limiting manner.

As used herein, the term “telescope” is intended to mean an instrumentdesigned for the observation of remote objects, typically comprising aninput optical component at the distal end of a tube and one or moreoptical components at the proximal end, such tube providing a path forlight beams and possibly a path to change the spacing of opticalcomponents and hence focus or magnification, including withoutlimitation a tubular optical instrument for viewing distant objects bymeans of the refraction of light rays through a lens or the reflectionof light rays by a concave mirror.

As used herein, the term “vergence” is intended to mean the ability ofthe optical axes of the eyes or of an optical instrument to rotatetoward or away from each other to remain pointed at an object as itapproaches or moves away.

As used herein, the term “view” is intended to mean extent or range ofvision.

Attention is now directed to more specific details of embodiments thatillustrate but not limit the invention.

General Description

One embodiment of the invention provides a stereoscopic camera thatincludes a Stereoscopic Image Acquisition Device (SIAD) having anacquisition controller and an optical instrument that may be attached toor built into the SIAD.

FIGS. 1-3 illustrate three embodiments of the invention having a varietyof components and applications. FIG. 1 illustrates one embodiment of theStereoscopic Camera and System and its components, and a viewer. Thecomponents are labeled as outlined below:

1. Object in a location being imaged;

2. Optical axes;

3. User interface, in this embodiment a pistol-grip with controls;

4. Stereoscopic Image Acquisition Device (SIAD);

5. Optical instrument, in this embodiment it is a binocular having twotelescopes that is attached to the SIAD;

6. Stereoscopic display, in this embodiment an autostereoscopic flatpanel;

7. Image processing unit (IPU), in this embodiment a display controlleris included in the IPU;

8. Viewer, in approximate viewing position in this embodiment to seeboth object and 3D image; and

9. Image of object in location, appearing in 3D to the viewer.

FIG. 2 shows a close-up of several components of the embodiment shown inFIG. 1 which are labeled as outlined below:

2. Optical axes;

3. User interface, in this embodiment it is a pistol grip with triggerfor image capture and thumb-operated joystick for other input functions;

4. Stereoscopic Image Acquisition Device. In this embodiment theconnections (not shown) from sensors to acquisition controller areflexible circuits, allowing mutual movement between sensors andacquisition controller;

5. Optical instrument, in this embodiment is a binocular having twotelescopes, (5 a) and (5 b) respectively;

6. Stereoscopic display, in this embodiment for images and userinterface;

7. IPU; in this embodiment a display controller is included in the IPU;

10 a and 10 b. Means to adjust the convergence of the optical axes, inthis embodiment it is a flexing beam (10 a) with a manually-driven screw(10 b);

11 a and 11 b. Means to adjust the interpupillary distance (IPD), inthis embodiment it is a slider traveling on a rail (11 a), controlled byan electromechanical actuator (11 b);

12. Means to adjust the vertical alignment of the cameras' images, inthis embodiment it is an actuator-driven screw; and

13. Battery pack.

Components

In one embodiment, the invention provides a stereoscopic camera systemthat includes a Stereoscopic Image Acquisition Device 4 (SIAD), anoptical instrument 5 that may be attached to or built into the SIAD, anda display mechanisms for displaying stereoscopic images 6 generated bythe optical instrument and SIAD. In yet another embodiment, the systemmay include an image processing unit 7 (IPU). In another embodiment, thesystem may include a battery or other power source 13 to provide powerto the system. Further embodiments may contain no SIAD but may containother components to perform similar functions.

The image processing unit 7 as well as the display 6 and a battery 13(to provide power) may be attached to the device and integrated into ahousing, resulting in a complete, integrated, one-piece device thatprovides all the necessary functionality including: mechanisms for (1)forming a stereoscopic electronic image of an object or location, (2)processing such image data and (3) displaying a magnified, unmagnified,or demagnified stereoscopic image of the object or location, in adesired orientation and condition (e.g. not inverted or reflected, or inany orientation desired by the user), in a convenient position on thestereoscopic display for the viewer, in real time, and in a device whichcould be portable.

In other embodiments, some or all of the functions of the IPU can bebuilt into circuitry, firmware and software inside the SIAD or elsewherein the system, such that a separate IPU component may not be required,possibly reducing the size and cost of the system. In yet otherembodiments, the one-piece device could be handheld in its use. Infurther embodiments, the display could be mounted separately with atethering data cable or wireless link to the SIAD or IPU.

In one embodiment, the display could be mounted to the user, facing theuser's view, such that his hands can be free to steer and operate thedevice or perform other tasks while observing the image on the display.An additional advantage of such a system is that the user can possiblysee the displayed image at the same time as his peripheral vision allowshim to see the rest of the location or surroundings, or vice-versa.Additionally. the system could have the display mounted directly on ahandheld device that may be the SIAD, to accomplish a similar result. Inthese or other embodiments the display, or one or more additionaldisplays connected via a “splitter” device, could be mounted such thatmultiple viewers could see the displayed image.

Image Processing

Image processing could be performed on the data to reduce the perceivedeffect of camera shake on the viewed stereoscopic image. Time sequentialdata from both left and right channels in combination could be used tocalculate corrections of the data to negate the shake effect of bothchannels, providing image stabilization for the entire stereoscopicimage for example, or to otherwise cause a desired effect. Because theelectronic corrections can be calculated knowing the kinematicrelationship between the optical axes and because the data from thesensors is synchronized with each other the proper corrections can beapplied to the stereoscopic image. This is advantageous as compared tothe prior art, which applied the corrections to the views separately andasynchronously, resulting in spurious parallax of the stereoscopicimage. Alternatively the corrections could be applied to one or moreactuation mechanisms to alter the optical path or paths, such that theimage is corrected when it arrives at the sensor.

Data from both left and right channels could be used to calculatecorrections to the sensor exposure parameters or corrections to the dataitself to optimize the stereoscopic image and to balance the leftchannel with the right, providing simultaneous exposure or gain controlfor example. One embodiment of this could involve a simultaneousbaseline setting of the two channels to give equal imagecharacteristics, for example performing white balance simultaneously.

Image processing to compress or encode the data could be done to thesingle data stream or to one or both streams prior to their combination.

Displays

In these and other embodiments of the current invention, thestereoscopic display could be of any type as described in the relatedU.S. Patent Application Ser. Nos. 60/762,577, 60/833,117, and U.S.patent application Ser. Nos. 11/256,497 and 11/881,617 that can befurther applicable to embodiments of this invention. In embodimentswhere the user looks slightly downward to see the display but looks upover the display to see the location or object being imaged, and wherethe display is of a type requiring the user to wear spectacles to seethe image stereoscopically, it could be advantageous for the spectaclesto be constructed like “reading glasses” whereby looking over the activeportion of such spectacles the user has an unobstructed view of thelocation. Alternatively, left-right stick-on films made from polarizers,retarders or other materials required for stereoscopic viewing of the 3Ddisplay, for viewers wearing other glasses, or polarized sunglasses,could be used. A graphic on the film could indicate proper orientation.A permanent or non-permanent adhering method could be used.

Another embodiment may have a display that could be switched fromdisplaying either left or right or both views. Another embodiment mayhave a display that presents either or both views together and has noprovision for stereopsis.

Storage

In other embodiments one or more Hard Disk Drives (HDD), Digital VideoDisks (DVD), solid-state or other similar data storage devices (SD),could be included in the IPU or elsewhere for storage, furtherprocessing and playback of images. Because the SIAD inputs, processesand displays images in generally real-time (as observed by a user), theSD interface circuitry could be constructed such that images can bestored essentially instantaneously, with no perceived latency after astart trigger has been activated. This has previously been a problemwith digital cameras, whereby a shutter button is pressed but the imageis not taken until after a noticeable delay. In some embodiments, thefile structure may be split into parallel channels such that thebandwidth of data is split to allow that flowing to each SD to be withinthe SD's transfer rate. In other embodiments, the storage system couldmake the actual recording of the images on the SD with a delay from whenthey are captured in system volatile memory. In yet other embodiments,streaming full-motion stereoscopic image data continuously through theSD system, such that there exists some quantity of image data previouslystored, upon the occurrence of an event desirable to be viewed butunanticipated, would allow a viewer to review such previously-storedstereoscopic images to view the event after its occurrence. Stereoscopicimage data could be updated continuously or in some other manner to, forexample, provide a time-lapse stereoscopic recording of an event thatoccurs slowly. In embodiments where the stereoscopic image data is asingle stream, additional synchronization may be unnecessary, resultingin a simpler system, both in storing images and playing them back. Sucha system generally has the same data integrity for both the left andright views of images, as opposed to dual-data-channel systems in whicheither channel may have defects that could spoil the stereoscopic image.Additionally, such a system could also be made redundant and fail-safe,and this could be done more easily to avoid any image data degradation.Such a system could use SDs that generally have low or no recurringcosts associated with re-saving newer image data over old. The SD, orits storage media, could be removable, replaceable or expandable, withpower on or off. The SD could be of low power such that it could beincluded in a battery-operated embodiment. The SD could be of low sizeand weight such that it could be included in a handheld embodiment. TheSD could be of sufficient robustness such that it could be included inan embodiment intended for severe-environments.

Lenses

Lenses or other optical components of the camera and their mountingcould be such that interchangeable lenses could be used. Such lensescould be capable of focusing light of infrared, UV or visiblewavelengths or combinations thereof. The camera could use two lenses,one for each L/R channel, or a single lens with an optical pathswitching device such as a shuttering system and beamsplitter or acombination of these. For underwater or similar applications, a systemwherein the lenses could contact the water directly and there is a sealfor the electronic portion of the SIAD could be used. Furthermore, theinternal volume of the sealed portion of the SIAD, possibly includingthe lenses, could be minimized and securely sealed such that noadditional outside housing would be required for use underwater,resulting in a small overall size and weight and easier use. Yet furtherthere could be a tether, thus forming a completely, possibly permanentlysealed camera head with a cable, optical fiber or wireless link to theIPU or Display. Such sealing could be potting compound.

The lenses could also be parabolic mirrors to focus directly on theimage sensors such as, for example, lenses useful for an IR imagingdevice with no IR glass optics. Alternatively, lenses could be focusedautomatically and/or together as described in the referenced patentapplications. In addition, the lenses could be small to be mountedclosely together to make a small inter-pupillary distance.

Convergence

In embodiments having two optical axes, these axes may be parallel, theymay converge at some point distant from the camera or they may beadjustable to converge at any point from a close distance to infinity orto diverge. To allow relative motion of the sensors in embodiments whereone or more of the sensors can be moved to move the optical axes, thesensors could be connected to the acquisition controller with flexiblecircuitry, which may be shielded, to allow relative motion whilemaintaining bandwidth and suppressing noise.

Convergence could be adjusted by use of a mechanism with an actuator todeflect either or both of the sensors' mounting and their respectivelens and axis. Alternatively the mechanism could move an opticalelement, for example a wedge lens, to deflect one or both axes. Themechanism could be comprised of a hinge or a flexing beam of metal,plastic or other suitable material. The actuator could be a manual screwor motorized leadscrew or cam system, or other mechanical orelectromechanical device. The control of the actuator could be designedto be operated by the camera user while shooting. The control could be aknob or switch that could also be a handle for one hand. The controlcould also be a pistol grip device where the user squeezes or activatesa lever or wheel to control convergence adjustment. Adjustable stops ordetents in the mechanism could be incorporated to aid the user toachieve their desired convergence effect. Convergence could be automatedto the autofocus or other aspects of the camera. The IPU, for example,could be programmed to recognize some moving feature and follow it withconvergence. There could be a second flexure or mechanism to adjust theaxes to be coplanar, known in the art as vertical alignment.

Embodiments of the camera or camera systems may also contain aconvergence mechanism such that the user can change the convergence asthe location or object is being observed and/or recorded. In suchembodiments the convergence mechanism can be constructed such that itcauses a continuous change in convergence, a smooth transition that iswithout abrupt changes in the views.

The convergence device and system described herein could also be usedwith two separate cameras and camera systems, for example, without theuse of the SIAD.

Inter-Pupillary Distance (IPD)

In some embodiments having two optical axes, it may be desirable tochange the distance between the axes as measured at the camera, known inthe art as the IPD. This could be done by use of a mechanism such as abeam that moves one sensor and its respective lens and housing away fromthe other sensor, generally perpendicular to the optical axes, thatmovement taking place in the plane of the axes. A clamp could be used tomanually secure the sensor housing along a beam in the desired locationto achieve the desired IPD. Such a beam could also be used to adjustconvergence by flexing or deflecting. Alternatively the IPD could beadjusted by use of a mechanism with an actuator to move either or bothof the sensors and their respective lens and axis. Such an embodimentcould be designed to accommodate interchangeable lenses includingwithout limitation wide angle or telephoto lenses for either or bothoptical channels.

Zoom

Zoom is known in the art as changing the magnification of an opticalchannel while a location is being observed. Zooming “in” is generallyincreasing the magnification and zooming “out” is generally decreasingthe magnification. Zooming can be performed manually through the use ofa mechanism or automatically by sending an electronic signal to anelectromechanical actuator. Either technique causes, for example,changes in the axial spacing between or shape of optical elements.Generally it is desirable to have zooming performed such that abruptchanges in views do not occur.

The mechanism could be geared to drive rotational cam devices thatappropriately change the spacing between optical elements, and hence themagnification, in each optical path simultaneously. The mechanism couldbe mechanical or electromechanical.

Digital zoom is an electronic method to present a similar effect ofincreasing magnification to a viewer of an electronic display. It istypically implemented by causing a subset of the pixels within the datarepresenting a view to be expanded in order to represent the entire dataset representing the view. In this approach, additional pixels arefabricated from adjacent pixel data and interspersed between theoriginal pixels to form a view having generally the same number ofpixels as the original view. Alternatively, digital zoom can beimplemented by choosing a subset of the pixel cells illuminated by thechannel optics on a sensor, to form the data representing the view fromthe sensor of the channel.

The embodiments of the invention may also contain electronic mechanismsto cause digital zoom to be applied to two channels simultaneously.Alternatively, zooming could be done on a master-slave basis whereby themaster view's (the right view in this embodiment) magnification is setas desired, and the slave view's (the left view in this embodiment) zoomis automatically controlled to match the master. Such control could usethe image data to calculate the proper control parameters, for exampleby measuring the number of pixels between certain features common ineach L/R view and attempting to equalize them, and could be a functionof the IPU or acquisition controller.

The zoom device and system described herein could also be usedindependently with two separate cameras and camera systems, for example,without the use of the SIAD.

Controls

In some embodiments the controls and functions could enable the deviceto be operated by a single person, remotely or automatically. Control ofone or more of the system's functions could be achieved through one ormore separate user interface devices used in combination, includingwithout limitation, pushbuttons or other switching devices, a touchpadscreen (separate or attached or a part of the 3D display), a joystick,pistol-grip control device, wheel, lever, mouse or similar devicecontrolling the system that provides user feedback on the 3D displayscreen or on another screen or output indication device. Such userinterface and feedback could be stereoscopic to enhance effectiveness.Such controls could operate remotely from the device via a wireless linkor over an interface.

Communication

Communication, including transmittal of image data, and additionalcontrol of a system of these embodiments could be performed thru anexternal interface, including without limitation a network, USB,“firewire”, cameralink or custom designed interface or port, to otherdevices or to a network.

Measurements

Capture of the stereoscopic image data could include the capability tomake measurements in three dimensions, x-y-z, z being into/out of theimage plane, using the parallax data between left/right views of anypoint common to both views, to calculate the z dimension. Furthermore,it may be desirable to calculate the location of points with respect toa coordinate system. An embodiment could include a component fordetermining the location of the stereoscopic camera with respect to thedesired coordinate system, for example a GPS receiver, whereby thecoordinate system is earth's latitude, longitude and altitude. Inaddition the embodiment could include a component for determining thedirection of the z dimension with respect to the coordinate system, forexample an electronic compass and/or an azimuth indicator. The locationof points in the stereoscopic image can then be calculated with respectto the coordinate system. Quantization error of such a system could bereduced by networking more than one system viewing the desired pointsfrom different camera locations and averaging their calculated locationsof the desired points.

An embodiment could include a pan-tilt-zoom mechanism and such mechanismcould be equipped with, for example, position indication devices suchthat the direction of the z dimension could be changed and the newdirection calculated with respect to the desired coordinate system.

Deflecting Element

FIG. 3 illustrates one embodiment of the Stereoscopic Camera and Systemcontaining a deflecting element, system components, and a viewer. Thecomponents are labeled as outlined below:

1. Object in a location being imaged;

2. Optical axes;

4. Stereoscopic Image Acquisition Device;

5. Optical instrument, in this embodiment is a binocular with twotelescopes;

6. Stereoscopic display, in this embodiment an autostereoscopic flatpanel.

8. Viewer, in approximate viewing position in this embodiment to see 3Dimage;

9. Image of object in location, appearing 3D to the viewer;

14. Deflecting element, in this embodiment a flat mirror, mounting notshown; and

15 a-15 c. Wall (15 a) between harsh environment (15 b) and non-harshenvironment (15 c); wall is shown in cutaway.

16. Pan-tilt-zoom mechanism for changing the direction and/ormagnification of the camera.

In this embodiment, a deflecting element 14 can be placed in the opticalpath 2 between the object being imaged 1 and the input optical componentof the optical instrument 5. A system with a deflecting element 14 couldbe used, as a nonlimiting example, to see around corners or to protectthe camera or camera system by exposing only the deflecting element 14to an environment unsuitable for the camera 15 b while keeping most ofthe system in a safe environment 15 c, protected from the unsafeenvironment 15 b by, for example, a suitable type of wall 15 a.Electronic mechanisms can be utilized to de-reflect the views or thestereoscopic image such that the viewer 8 observes an image 9 of thelocation or object as if it was not reflected by the deflecting element.Alternatively, the deflecting element can be adjusted to provide otherdesirable orientation.

The embodiments and examples set forth herein, including the best modeknown to the inventor for carrying out the invention, were presented inorder to best explain embodiments of the present invention and itspractical application and to thereby enable those of ordinary skill inthe art to make and use embodiments of the invention. However, those ofordinary skill in the art will recognize that the foregoing descriptionand examples have been presented for the purposes of illustration andexample only. The description as set forth is not intended to beexhaustive or to limit the invention to the precise form disclosed. Manymodifications and variations are possible in light of the teachingsabove without departing from the spirit and scope of the forthcomingclaims. Furthermore, embodiments of the present invention are alsoapplicable to many other types of optical instruments.

I claim:
 1. An apparatus for acquiring at least one stereoscopic imageof an object, the apparatus comprising: (a) at least one image sensorcomprising pixel cells configured to generate pixel data; (b) at leastone first optical instrument comprising at least one channel, the atleast one channel comprising an optical path and at least one opticalcomponent wherein the at least one optical component is configured toilluminate at least a portion of the pixel cells of the at least oneimage sensor, wherein the at least one optical instrument is capable ofproviding a plurality of optical views corresponding to a left view anda right view of at least one stereoscopic image of the object; (c) anacquisition controller configured to acquire from the at least onesensor at least one pixel data for each view of the plurality of opticalviews wherein the at least one pixel data corresponding to the left viewis simultaneously generated with the at least one pixel datacorresponding to the respective right view; (d) a data processorconfigured to combine the at least one pixel data for each view of theplurality of optical views into a single data stream and transmit, via afirst interface, the single data stream, in real time.
 2. The apparatusof claim 1, further comprising a display controller configured toreceive the single data stream, process the single data stream intodisplay signals and send the display signals to at least one display. 3.The apparatus of claim 2, further comprising an electronic or mechanicalor electromechanical mechanism for adjusting a vertical position of atleast a portion of the object in at least one view of the plurality ofoptical views.
 4. The apparatus of claim 1, wherein the data processoris further configured to combine the at least one pixel data for eachview of the plurality of optical views into the single data streambefore storage of the at least one pixel data corresponding to at leastone view of the plurality of optical views.
 5. The apparatus of claim 1,wherein the acquisition controller is further configured to changeexposure parameters in the at least one image sensor.
 6. The apparatusof claim 1, further comprising a storage device configured to receivethe transmitted single data stream and store it in a single datastructure.
 7. The apparatus of claim 1 wherein the at least one imagesensor and the at least one optical instrument are further configured togenerate the pixel data from illumination in at least one range ofwavelengths selected from the list of ultraviolet, visible, andinfrared.
 8. The apparatus of claim 1 wherein the first interface is ahigh-speed data pathway.
 9. The apparatus of claim 8 wherein the atleast one pixel data for each view of the plurality of optical viewscomprising the single data stream is transmitted to and processed byanother device, and the timing of the processed at least one pixel datafor each view of the plurality of optical views of the single datastream is maintained.
 10. The apparatus of claim 1 further comprising apan-tilt-zoom mechanism comprising devices for indicating the positionand direction of the apparatus.
 11. The apparatus of claim 1 furthercomprising a first coupling mechanism, the coupling mechanism supportingthe apparatus relative to a second optical instrument, and automaticallyaligning at least one output optical path of the second opticalinstrument with the at least one channel of the first opticalinstrument.
 12. The apparatus of claim 11 further comprising a secondcoupling mechanism adapted for supporting an optical viewing module, andautomatically aligning at least one input optical path of the opticalviewing module with the at least one channel of the first opticalinstrument.
 13. The apparatus of claim 1, further comprising at leastone filter component and a replacement mechanism connected to theapparatus for replacing the at least one filter component.
 14. Anapparatus for acquiring stereoscopic images of a location, the apparatuscomprising: (a) at least one optical instrument comprising a first and asecond channel, the channels comprising a first and a second opticalaxis, each axis comprising a proximal end positioned at the approximatecenter of an array of pixel cells of at least one image sensor, whereinthe at least one optical instrument is capable of providing a pluralityof optical views corresponding to a first left view and a first rightview of a first stereoscopic image of the location to the at least oneimage sensor; (b) a convergence device for setting the first and thesecond respective optical axes of each respective optical channel,thereby aligning a vergence of distal intersections of the first andsecond respective optical axes to coincide at a first desired distance;(c) an acquisition controller configured to acquire, from the at leastone sensor, pixel data representative of each view of the plurality ofoptical views; (d) an image processing unit communicatively coupled tothe acquisition controller and configured to: i. receive the pixel datarepresentative of each view of the plurality of optical views of thefirst stereoscopic image of the location; ii. choose a first subset ofthe pixel data representative of at least a portion of the first leftview and a corresponding second subset of the pixel data representativeof at least a portion of the first right view wherein the first andsecond subsets comprise respective second left and second right views ofa second stereoscopic image of the location; and iii. output thesubsets.
 15. The apparatus of claim 14 further comprising a displaycommunicatively coupled to the image processing unit, the displayconfigured to display the first subset and the second subset of thepixel data.
 16. The apparatus of claim 15 wherein the pixel datacorresponding to the proximal ends of the respective first and secondoptical axes lie outside the approximate centers of the displayedrespective first and second subsets of the pixel data.
 17. The apparatusof claim 15 wherein the display is stereoscopic and the subsets aredisplayed as the second stereoscopic image.
 18. The apparatus of claim17 wherein the displayed second stereoscopic image is representative ofa magnification of at least a portion of the first stereoscopic image.19. The apparatus of claim 14 wherein the first desired distance isinfinity.
 20. The apparatus of claim 14 wherein at least one pixel datarepresentative of the first left view is simultaneously generated withthe respective at least one pixel data corresponding to the first rightview.